1. Technical Field
The present invention relates generally to RDMA (remote data memory access) systems, and more particularly relates to an asynchronous completion notification system for an RDMA network interface card.
2. Related Art
RDMA (remote data memory access) is a network interface card (NIC) feature that lets one computer directly place information into the memory of another computer. The technology reduces latency by minimizing demands on bandwidth and processing overhead. Traditional hardware and software architecture imposes a significant load on a server's CPU and memory because data must be copied between the kernel and application. Memory bottlenecks become more severe as connection speeds exceed the processing power and memory bandwidth of servers.
RDMA gets around this by implementing a reliable transport protocol in hardware on the NIC and by supporting zero-copy networking with kernel bypass. Zero-copy networking lets the NIC transfer data directly to or from application memory, eliminating the need to copy data between application memory and the kernel.
Kernel bypass lets applications issue commands to the NIC without having to execute a kernel call. The RDMA request is issued from user space to the local NIC and over the network to the remote NIC without requiring any kernel involvement. This reduces the number of context switches between kernel space and user space while handling network traffic.
The RDMA protocol is defined by the RDMA Consortium, which in part, maps the RDMA features of Infiniband onto Ethernet. The RDMA and InfiniBand standards provide the concept of a completion queue (CQ) for holding “consumer reports” about completion requests posted to the work (i.e., send or receive) queue. Each entry in the CQ is called a completion queue entry (CQE). The standards also provide the concept of an asynchronous completion notification mechanism, which is used to notify the consumer when a new CQE is placed in the CQ. In this mode of operation, the consumer can register an asynchronous completion notification handler, which is called when:                (1) the consumer requested asynchronous completion notification (by calling the request completion notification verb); and        (2) a new CQE is placed in the CQ (after the request completion notification verb was invoked).The CQE can be retrieved from the CQ using the poll for completion verb.        
As part of any efficient software implementation using an Asynchronous Completion Notification mechanism, the RNIC (remote network interface card) needs to guarantee that, given proper software behavior, no CQE will ever be left unattended in the CQ, i.e., each CQE placed to the CQ will either be retrieved by a Poll For Completion, or be indicated by a call to the Asynchronous Completion Notification Handler routine registered by the software.
Unfortunately, using known implementation techniques, situations may arise wherein one or more CQE's may be left unattended in the CQ. Accordingly, a need exists for an Asynchronous Completion Notification system that can guarantee that no CQE will ever be left unattended in the CQ.